PROMPT DESIGN

ZhuJiu-Knowledge: A Fairer Platform for Evaluating Multiple Knowledge Types in Large Language Models (2024.naacl-demo)

Challenge:	Existing prompt optimization methods often underperform due to learning exclusively from incorrect samples.
Approach:	They propose a framework that leverages contrastive prompts to distinguish between high- and low-performing cases.
Outcome:	The proposed framework can generalize across open and proprietary models and NLU benchmarks.

Copied to clipboard

Pengfan Du, Sirui Liang, Baoli Zhang, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

Thesis Proposal: Multimodal Benchmark for Music Understanding in Large Language Models (2026.eacl-srw)

Challenge:	evaluating the knowledge of large language models (LLMs) is crucial, and rapid advancement in large language modeling has heightened the importance of model evaluations.
Approach:	They propose a fairer benchmark for evaluating multiple knowledge types of LLMs by focusing on commonsense knowledge, world knowledge, and language knowledge.
Outcome:	The proposed framework evaluates 14 current mainstream LLMs and provides a detailed discussion and analysis of their results.

Copied to clipboard

Tomáš Sourada

SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Data (2023.findings-emnlp)

Challenge:	Existing music-focused benchmarks are fragmented, largely single-modality, Western-centric . existing methods for evaluating MLLMs are lacking reproducibility and reliability .
Approach:	They propose to develop a musically multimodal benchmark that will integrate music into the benchmark.
Outcome:	The proposed benchmark will integrate culturally diverse musical material beyond the dominant Western canon.

Copied to clipboard

Ruoxi Sun, Sercan Arik, Rajarishi Sinha, Hootan Nakhost, Hanjun Dai, Pengcheng Yin, Tomas Pfister

Programming Languages in NLP

Challenge:	Text-to-SQL aims to automate the process of generating SQL queries on a database from natural language text.
Approach:	They propose a method to improve few-shot prompting capabilities of Text-to-SQL for Large Language Models (LLMs) they propose 'SQlPrompt' which aims to diversify the SQL proposals during consistency selection with different prompt designs and foundation models.
Outcome:	The proposed method outperforms previous approaches for in-context learning with zero labeled data by a large margin, closing the gap with finetuning state-of-the-art with thousands of labeles.

Transforming Code Understanding: Clustering-Based Retrieval for Improved Summarization in Domain-Specific Languages (2025.coling-industry)

Copied to clipboard

Baban Gain, Dibyanayan Bandyopadhyay, Samrat Mukherjee, Aryan Sahoo, Saswati Dana, Palanivel Kodeswaran, Sayandeep Sen, Asif Ekbal, Dinesh Garg

Information Extraction & Text Mining

Challenge:	Existing natural language summaries of domain-specific languages are limited due to their recency and complexity.
Approach:	They propose a clustering-based technique to retrieve in-context examples that are semantically closer to the test example and propose eBPF prompt generation technique that yields superior-quality code summary generation.
Outcome:	The proposed method improves the eBPF code summarization accuracy by 12.9 BLEU points over other prompting techniques.

AutoBool: Reinforcement-Learned LLM for Effective Automatic Systematic Reviews Boolean Query Generation (2026.eacl-long)

Copied to clipboard

Shuai Wang, Harrisen Scells, Bevan Koopman, Guido Zuccon

Quantifying Aleatoric Uncertainty of In-Context Learning for Robust Measure of LLM Prediction Confidence (2026.acl-long)

Challenge:	Existing approaches to generate Boolean queries for systematic reviews are limited by the lack of ground-truth best Boolesan queries.
Approach:	They propose a reinforcement learning framework that trains large language models to generate effective Boolean queries for medical systematic reviews.
Outcome:	The proposed framework outperforms zero-shot/few-shot prompting on 65 588 topics . it also matches or exceeds the effectiveness of larger GPT-based models using smaller backbones .

Copied to clipboard

Jinseok Chung, Minkyoung Song, Hyunji Jung, Namhoon Lee

Metacognitive Prompting Improves Understanding in Large Language Models (2024.naacl-long)

Challenge:	Existing methods for standard generation tasks fail to capture the unique dynamics of ICL.
Approach:	They propose a concept of self-function vectors that leverage Bayesian views and the mechanistic interpretability of ICL to model latent concept learned during in-context prompting.
Outcome:	The proposed framework can be used for trustworthy-related applications, such as hallucination detection.

Copied to clipboard

Yuqing Wang, Yun Zhao

Zero-Shot Text Classification via Self-Supervised Tuning (2023.findings-acl)

Challenge:	Recent advances in prompting have enhanced reasoning in logic-intensive tasks for LLMs, yet the nuanced understanding abilities of these models remain underexplored.
Approach:	They propose a strategy inspired by human introspective reasoning processes to enhance LLMs' understanding abilities.
Outcome:	The proposed method outperforms chain-of-thought prompting and its advanced versions on ten natural language understanding (NLU) datasets.

Copied to clipboard

Chaoqun Liu, Wenxuan Zhang, Guizhen Chen, Xiaobao Wu, Anh Tuan Luu, Chip Hong Chang, Lidong Bing

Low-Resource NLP

Challenge:	Existing solutions to zero-shot text classification use pre-trained language models or large-scale annotated data.
Approach:	They propose a self-supervised learning paradigm to solve zero-shot text classification tasks by tuning the language models with unlabeled data.
Outcome:	The proposed model outperforms the state-of-the-art models on 7 out of 10 tasks and is less sensitive to prompt design.

GraphICL: Unlocking Graph Learning Potential in LLMs through Structured Prompt Design (2025.findings-naacl)

Copied to clipboard

Yuanfu Sun, Zhengnan Ma, Yi Fang, Jing Ma, Qiaoyu Tan

Deploying Tiny LVLM Judges for Real-World Evaluation of Chart Models: Lessons Learned and Best Practices (2025.emnlp-industry)

Challenge:	Text-Attributed Graphs (TAGs) are a powerful tool for understanding complex systems and relationships.
Approach:	They propose a benchmark to evaluate large language models for graph-structured data using prompts.
Outcome:	The proposed benchmark outperforms state-of-the-art graph LLMs and graph neural networks on graph learning tasks without training.

Copied to clipboard

Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Mizanur Rahman, Amran Bhuiyan, Israt Jahan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, Jimmy Huang

SocREval: Large Language Models with the Socratic Method for Reference-free Reasoning Evaluation (2024.findings-naacl)

Challenge:	Large Vision-Language Models (LVLMs) with only 7B parameters perform poorly as judges in resource-constrained settings.
Approach:	They propose two approaches to ensure costefficient evaluation by combining multiple criteria into a single query and domainadaptive transfer learning to create a 2Bparameter VLM on a chart dataset.
Outcome:	The proposed model can effectively transfer knowledge from one dataset to another to make it a more specialized model.

Copied to clipboard

Hangfeng He, Hongming Zhang, Dan Roth

M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning (2024.emnlp-main)

Challenge:	Existing reference-free reasoning evaluation metrics rely on human-annotated reasoning chains as references, but require fine-tuning with human-derived chains before evaluation.
Approach:	They propose to use GPT-4 to automatically evaluate reasoning chain quality by leveraging the Socratic method.
Outcome:	Empirical results show that the proposed approach significantly improves existing reference-free reasoning evaluation metrics.

Copied to clipboard

Taowen Wang, Yiyang Liu, James Liang, Junhan Zhao, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, Lifu Huang, Qifan Wang, Dongfang Liu

ThinkSwitcher: When to Think Hard, When to Think Fast (2025.findings-emnlp)

Challenge:	Multimodal Large Language Models (MLLMs) exhibit remarkable performance across a wide range of domains.
Approach:	They propose a multimodal prompt tuning approach for efficient instruction tuning of MLLMs.
Outcome:	The proposed approach shows superior performance on multimodal evaluation datasets compared to state-of-the-art methods.

Copied to clipboard

Guosheng Liang, Longguang Zhong, Ziyi Yang, Xiaojun Quan

CoE-SQL: In-Context Learning for Multi-Turn Text-to-SQL with Chain-of-Editions (2024.naacl-long)

Challenge:	Large reasoning models excel at solving complex tasks by leveraging long chain-of-thought (CoT) reasoning.
Approach:	They propose a framework that enables a single LRM to dynamically switch between short and long CoT modes based on task complexity.
Outcome:	The proposed framework reduces computational cost by 20-30% while maintaining high accuracy on complex tasks.

Copied to clipboard

Hanchong Zhang, Ruisheng Cao, Hongshen Xu, Lu Chen, Kai Yu

Programming Languages in NLP

Challenge:	Recent studies have demonstrated that Large Language Models (LLMs) have impressive capabilities in a variety of domains and tasks.
Approach:	They propose a method which prompts LLMs to generate SQL queries based on the previously generated SQL query with an edition chain.
Outcome:	The proposed method outperforms different in-context learning baselines and achieves state-of-the-art performance on two benchmarks SParC and CoSQL using LLMs.

Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning (2024.emnlp-main)

Copied to clipboard

Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, Meng Jiang

Decorate the Examples: A Simple Method of Prompt Design for Biomedical Relation Extraction (2022.lrec-1)

Challenge:	Experimental results demonstrate that OPPU significantly outperforms existing prompt-based methods across seven diverse tasks in the LaMP benchmark.
Approach:	They propose to integrate parametric user knowledge into the personal PEFT parameters and non-parametric knowledge from retrieval and profiles, adapting LLMs to user behavior shifts.
Outcome:	The proposed method outperforms existing prompt-based methods across seven diverse tasks in the LaMP benchmark.

Copied to clipboard

Hui-Syuan Yeh, Thomas Lavergne, Pierre Zweigenbaum

Unanswerability Evaluation for Retrieval Augmented Generation (2025.acl-long)

Challenge:	Recent research shows that prompt-based learning improves performance on relation extraction tasks.
Approach:	They propose a prompt-based learning method that generates comprehensive prompts for biomedical relation extraction using a ChemProt dataset.
Outcome:	The proposed method improves fine-tuning on a biomedical relation extraction task with a cloze-test task and fewer training examples to make reasonable predictions.

Copied to clipboard

Xiangyu Peng, Prafulla Kumar Choubey, Caiming Xiong, Chien-Sheng Wu

Enhancing Retrieval-Augmented Generation: A Study of Best Practices (2025.coling-main)

Challenge:	Existing evaluation frameworks for retrieval-augmented generation (RAG) systems focus on answerable queries, but ignore the importance of appropriately rejecting unanswerable requests.
Approach:	They propose a framework to evaluate whether retrieval-augmented generation systems handle unanswerable queries specific to a given knowledge base.
Outcome:	The proposed framework synthesizes diverse and challenging queries for any given knowledge base and evaluates them with unanswered ratio and acceptable ratio metrics.

Copied to clipboard

Siran Li, Linus Stenzel, Carsten Eickhoff, Seyed Ali Bahrainian

Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction (2022.acl-long)

Challenge:	Retrieval-augmented generation systems have shown remarkable advancements by integrating retrieval mechanisms into language models, enhancing their ability to produce more accurate and contextually relevant responses.
Approach:	They propose to integrate query expansion, various novel retrieval strategies, and a Contrastive In-Context Learning RAG to improve response quality.
Outcome:	The proposed RAGs incorporate query expansion, various novel retrieval strategies, and a novel Contrastive In-Context Learning RAG.

Copied to clipboard

Yubo Ma, Zehao Wang, Yixin Cao, Mukai Li, Meiqi Chen, Kun Wang, Jing Shao

Emotion Recognition in Conversation via Dynamic Personality (2024.lrec-main)

Challenge:	Using a prompt-based model, we find that event argument extraction is efficient and generalized well to few-shot settings.
Approach:	They propose a model PAIE for event argument extraction using prompt tuning for extractive objectives.
Outcome:	The proposed model can extract arguments with the same role instead of heuristic threshold tuning.

Copied to clipboard

Yan Wang, Bo Wang, Yachao Zhao, Dongming Zhao, Xiaojia Jin, Jijun Zhang, Ruifang He, Yuexian Hou

Dialogue Systems & Conversational Agents

Challenge:	Existing approaches to ERC focus on conversational contexts, but focus on static personality.
Approach:	They propose a model that considers the dynamic personality of speakers during conversations.
Outcome:	The proposed model outperforms existing models on three benchmark conversational datasets.

‘Hello, World!’: Making GNNs Talk with LLMs (2025.findings-emnlp)

Copied to clipboard

Sunwoo Kim, Soo Yong Lee, Jaemin Yoo, Kijung Shin

Structured Data in NLP

Challenge:	graph neural networks have shown remarkable performance across diverse graph-related tasks, but their high-dimensional hidden representations render them black boxes.
Approach:	They propose a graph-based neural network with hidden representations in the form of human-readable text.
Outcome:	The proposed GNN outperforms existing LLM-based baseline methods on node classification and link prediction.

VMLU Benchmarks: A comprehensive benchmark toolkit for Vietnamese LLMs (2025.acl-long)

Copied to clipboard

Cuc Thi Bui, Nguyen Truong Son, Truong Van Trang, Lam Viet Phung, Pham Nhut Huy, Hoang Anh Le, Quoc Huu Van, Phong Nguyen-Thuan Do, Van Le Tran Truc, Duc Thanh Chau, Le-Minh Nguyen

Challenge:	The evolution of Large Language Models (LLMs) has underscored the need for benchmarks designed for various languages and cultural contexts.
Approach:	They propose to use Vietnamese multitask language understanding (VMLU) benchmarks to assess different capabilities of LLMs, including general knowledge, reading comprehension, reasoning, and conversational skills.
Outcome:	The VMLU Benchmarks assess LLMs' general knowledge, reading comprehension, reasoning, and conversational skills.

APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models (2023.emnlp-main)

Copied to clipboard

Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, Dongfang Liu

AMPERE: AMR-Aware Prefix for Generation-Based Event Argument Extraction Model (2023.acl-long)

Challenge:	Existing prompt tuning methods only introduce prompts at the input layer, limiting performance and leaving large room for improvement.
Approach:	They propose a method that involves tuning a small set of soft prompts for pre-trained language models.
Outcome:	The proposed method outperforms state-of-the-art methods with pre-trained models on the SuperGLUE benchmark.

Copied to clipboard

I-Hung Hsu, Zhiyu Xie, Kuan-Hao Huang, Prem Natarajan, Nanyun Peng

Information Extraction & Text Mining

Challenge:	Existing generation-based EAE models focus on problem re-formulation and prompt design without incorporating additional information that has been shown to be effective for classification-based models.
Approach:	They propose to incorporate AMR into generation-based EAE models by generating AMR-aware prefixes for every layer of the generation model.
Outcome:	The proposed model generates AMR-aware prefixes for every layer of the generation model and improves the generation.

MixPAVE: Mix-Prompt Tuning for Few-shot Product Attribute Value Extraction (2023.findings-acl)

Copied to clipboard

Li Yang, Qifan Wang, Jingang Wang, Xiaojun Quan, Fuli Feng, Yu Chen, Madian Khabsa, Sinong Wang, Zenglin Xu, Dongfang Liu

Filter-then-Generate: Large Language Models with Structure-Text Adapter for Knowledge Graph Completion (2025.coling-main)

Challenge:	Existing methods for product attribute value extraction focus on extracting values for a set of known attributes with sufficient training data.
Approach:	They propose a prompt tuning approach to extract attributes from product information using mixed prompts.
Outcome:	The proposed approach improves on two product benchmarks and shows parameter-efficient training and avoids model overfitting.

Copied to clipboard

Ben Liu, Jihai Zhang, Fangquan Lin, Cheng Yang, Min Peng

AnchorCoT: Anchors Pave the Way for Multi-hop Reasoning (2025.findings-acl)

Challenge:	Empirical evidence suggests that LLMs perform worse than conventional KGC approaches.
Approach:	They propose a filter-then-generate paradigm and a multiple-choice question format to harness the capability of LLMs while mitigating the issue casused by hallucinations.
Outcome:	The proposed method achieves substantial performance gain compared to existing state-of-the-art methods.

Copied to clipboard

Tianshi Ming, Xian Wu, Yingying Zhang, Zichuan Fu, Dawei Cheng

Natural Language Interfaces

Challenge:	Large language models (LLMs) have demonstrated potential reasoning capabilities through prompt design, such as the Chain of Thought (CoT).
Approach:	They propose a new reasoning approach that predicts key entities which work as important “anchors” and employs a ranking algorithm to ensure the logical sequence of the predicted answers.
Outcome:	The proposed approach outperforms existing methods in multi-hop question reasoning and provides more accurate reasoning results in multihop question answering tasks.

Uncertainty Calibration for Tool-Using Language Agents (2024.findings-emnlp)

Copied to clipboard

Hao Liu, Zi-Yi Dou, Yixin Wang, Nanyun Peng, Yisong Yue

Explain-then-Process: Using Grammar Prompting to Enhance Grammatical Acceptability Judgments (2025.findings-acl)

Challenge:	Language agents are increasingly used to perform tasks and interact with a variety of external tools to achieve specific, goal-oriented objectives.
Approach:	They propose a tool calibration tool called ProbeCal which recalibrates the internal probabilities of tool-using language agents to better reflect the actual effectiveness of tool.
Outcome:	The proposed model significantly improves off-the-shelf language models in tool-using applications.

Copied to clipboard

Russell Scheinberg, Ameeta Agrawal, Amber Shore, So Young Lee

Bandit-Based Prompt Design Strategy Selection Improves Prompt Optimizers (2025.findings-acl)

Challenge:	Large language models (LLMs) can explain grammatical rules, but fail to apply those rules when judging sentence acceptability.
Approach:	They propose a grammar prompting paradigm that feeds an LLM's metalinguistic explanation back to the target model before deciding which sentence of a minimal pair is grammatical.
Outcome:	The proposed model improves on the English BLiMP, Chinese SLING, and Russian RuBLimp benchmarks.

Copied to clipboard

Rin Ashizawa, Yoichi Hirose, Nozomu Yoshinari, Kento Uchida, Shinichi Shirakawa

TABARD: A Novel Benchmark for Tabular Anomaly Analysis, Reasoning and Detection (2025.findings-emnlp)

Challenge:	Existing prompt optimization methods have found effective prompts, but they often differ from sophisticated prompts carefully designed by human experts.
Approach:	They propose to integrate prompt design strategies into prompt optimization by using a Thompson sampling-based approach.
Outcome:	The proposed method incorporates prompt design strategies into the prompt optimization process.

Copied to clipboard

Manan Roy Choudhury, Anirudh Iyengar Kaniyar Narayana Iyengar, Shikhhar Siingh, Sugeeth Puranam, Vivek Gupta

Unveiling Privacy Risks in LLM Agent Memory (2025.acl-long)

Challenge:	a new benchmark is constructed to evaluate the accuracy of large language models for tabular data . the benchmark uses direct, indirect, and Chain-of-Thought prompting .
Approach:	They propose a framework that uses prompting, self-verification and constraint-based rule execution to improve accuracy.
Outcome:	The proposed framework significantly improves accuracy and recall in tabular data.

Copied to clipboard

Bo Wang, Weiyi He, Shenglai Zeng, Zhen Xiang, Yue Xing, Jiliang Tang, Pengfei He

Promptception: How Sensitive Are Large Multimodal Models to Prompts? (2025.findings-emnlp)

Challenge:	Large Language Model (LLM) agents store private user-agent interactions in memory for demonstrations, introducing new privacy risks for LLM agents.
Approach:	They propose an attack that extracts private information from memory under a black-box setting and propose a method that can be used to attack the agent.
Outcome:	The proposed attack is effective under a black-box setting and it is demonstrated on two representative agents.

Copied to clipboard

Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, Salman Khan

Dissecting Clinical Reasoning in Natural Language Inference for Large Language Models (2026.findings-acl)

Challenge:	despite the success of Large Multimodal Models, prompt design for MCQA remains poorly understood.
Approach:	They propose a framework for evaluating prompt sensitivity in LMMs . they propose 61 prompt types, each targeting specific aspects of prompt formulation .
Outcome:	The proposed framework evaluates 10 LMMs across 3 MCQA benchmarks.

Copied to clipboard

Mael Jullien, Andre Freitas, Marco Valentino, Leonardo Ranaldi

Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMs (2025.acl-long)

Challenge:	Recent studies on large language models (LLMs) have demonstrated the impact of prompting strategies and fine-tuning techniques on their reasoning capabilities.
Approach:	They examine four classes of prompting strategies to elicit reasoning in large language models . they then construct demonstrations using a frontier model to distil multi-step reasoning capabilities into smaller models based on Low-Rank Adaptation (LoRA).
Outcome:	The proposed model improves in 75% of the models on MedNLI and TREC Clinical Trials.

Copied to clipboard

Xiang Zhang, Juntai Cao, Chenyu You, Dujian Ding

Low-Resource NLP

Challenge:	Chain-of-Thought (CoT) prompting has emerged as a practical workaround, but most CoT-based methods rely on a single generic prompt like “think step by step” with no task-specific adaptation.
Approach:	They propose a theoretical framework that explains why some prompts succeed while others fail by using a generic generic prompt like "think step by step" they show that prompts function as selectors, extracting specific task-relevant information from the model's full hidden state during CoT reasoning.
Outcome:	The proposed framework explains why some prompts succeed while others fail.

Probing Social Identity Bias in Chinese LLMs with Gendered Pronouns and Social Groups (2026.findings-acl)

Copied to clipboard

Geng Liu, Li Feng, Junjie Mu, Mengxiao Zhu, Francesco Pierri